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ABSTRACT: 

In analog signal processing, knowledge is encapsulated in the form 
of analog synaptic weights. In simple hardware systems, these 
weights are determined by resistance values. The analog 
computation of a scalar product and the subsequent nonlinear 
mapping can be performed by a summing amplifier with saturation. 
In this paper, a design for a general- purpose programmable, current- 
mode neural network using VLSI circuits operated in a subthreshold 
mode is proposed. The proposed neuron circuit consists of a CMOS 
differential pair, and the synapse circuit uses current mirrors 
controlled by static RAM (SRAM). A 2-link robot forward 
kinematics has been implemented in our neural chip design. 

INTRODUCTION 

A significant research activity has been developed around artificial neural networks in 
the past two decades. From a practical point of view, VLSI implementation of neural 
networks is very important because neural networks simulated on a digital computer 
are not able to take an advantage of parallel processing. VLSI technology allows us 
not only to simulate biological systems, but also to emulate them in designing 
artificial sensory systems in real-time. Many approaches on artificial neural network 
implementations in VLSI technology have been reported in the literature (Watanabe 
et al. y 1993; Graf and Jackel, 1989; Atlas and Suzuki, 1989; Morishita et aL, 1990; 
Boahen et al., 1989). Bipolar transistors were used for reducing multiplier error 
based on a bipolar-MOS analog circuit (Morishita et aL, 1990). In our design, the 
MOS transistors are operated under subthreshold region to keep the power 
consumption low. An on-chip SRAM cell array is used to digitally store 8-bit 
synaptic weights in this paper. The digital memory is the most straightforward 
implementation of a quantized analog storage. Another advantage to digital circuits 
is that they provide for the high arithmetic precision requirements of many existing 
learning algorithms. In general, 8 and sometimes 16 bits of precision are necessary 
for the synaptic weight representation for hardware implementation. Wawrzynek et 
al. (1993) also showed that 16-bit weight values and 8-bit output values were 
sufficient to achieve training and classification results comparable to 32-bit floating- 
point operation . The digital storage using an SRAM provides several advantages 
over using analog storage on an SRAM. The first advantage is the easy 
programmability. The analog storage requires a highly sensitive voltage sensor 
circuitry since it divides the voltage of the cell node into 2" levels to store w-bit data. 
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In this paper, VLSI architecture for a subthreshold operation of a 
programmable general- purpose neural network is proposed. For a neuron 
(processing element) circuit, a CMOS differential pair is utilized. For a synapse 
circuit, current multipliers are utilized to implement digital binary weights. An on- 
chip SRAM cell array is included so that the weights can be programmed. 

The remainder of this paper is organized as follows. The next section 
provides the structure of the subthreshold CMOS neuron and synapse circuits in 
detail; including the operation of the network with simulation results of the neuron 
circuit. Then the following section illustrates an implementation of a 2-link robot 
forward kinematics as a case study of this network design and its experimental 
results- The final section concludes this paper with some possible future issues that 
need to be addressed. 

STRUCTURE OF A PROGRAMMABLE NEURAL NETWORK 

A block diagram for hardware implementation of the current-mode neural network is 
shown in Figure 1. This diagram shows a case with a single neuron network; 
however, one can construct and extend to any single- or multi-layered network by 
conserving this concept. The neuron circuit consists a differential pair with Wilson 
current mirrors to obtain current-mode, positive and negative activation functions as 
shown in Figure 2(a). A simple and unique current-to-voltage converter has been 
designed and utilized in the neuron circuit to convert the sum of currents from the 
synapse circuit cells in the previous layer (simply the input current for the input stage) 
to voltage that is applied to the gate of input node of the differential pair as shown in 
Figure 2(b). This current- to- voltage converter is also designed such that it avoids 
polysilicon resistors that would consume relatively large chip area. The reference 
voltage on the other transistor in the differential pair is set to 2.5V with the voltage 
source circuit shown in Figure 2(c). Notice that this is simply a voltage divider using 
an n-channel transistor and a p-channel transistor as resistors, and one can obtain any 
desired voltage-division ratio by choosing proper W/L ratios of the MOS transistors. 
The transfer characteristics of the unipolar sigmoid function is illustrated in Figure 3. 
In this design, both positive and negative sigmoid functions can be available; 
however, only the positive sigmoid function is used for a unipolar neuron. 

Figure 4 shows the circuit schematic of an 8-bit digital programmable synapse 
cell with a sign bit. The transistors M1-M8 are sized such that the W/L ratios are 2/2, 
4/2, 8/2, 16/2, 32/2, 64/2, 128/2, and 256/2 (|im) to obtain an 8-bit binary 
multiplication. These transistors can be programmed via an SRAM cell array so that 
one can obtain any 8-bit weights. The sign-bit controls the direction of the output 
current and is turned on for a negative current output. The transistors M10-M14 
have the same size; therefore, when the sign-bit is turned on, the drain current of 
M10 is copied to that of Ml 3 and Ml 4. By pulling down the drain current of Mil 
with twice as much using M13 and M14, the direction of the output current becomes 
opposite compared to a positive output current. Consequentiy, a negative current or 
a negative weight can be produced. In Figure 4 the direction of negative current flow 
is to the left. Notice that typical sigmoidal activation functions are 
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Figure 1: Block diagram of the current-mode neural network VLSI implementation. 
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Figure 2: Neuron circuit using MOS differential pair with Wilson current mirrors: 
(a) Neuron circuit, (b) Cur rent-to- voltage converter, and (c) 2.5V voltage source. 

All the transistors have W/L ratio of Z\xml1\xm. 




Figure 3: T insfer characteristics of the unipolar neuron circuit 
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where X > 0 and is proportional to the neuron gain determining the steepness of the 
continuous function / (net) near net = 0. Thus, instead of controlling the value of X 
in the sigmoidal activation function, it is possible to compensate the right-hand term 
of (1) by controlling the net value or by multiplying a certain fraction to the weights. 
This compensation can be accomplished by adjusting the W/L ratios of transistors M9 
and M10 in Figure 4. 

In some neural network applications, such as forward and/or inverse robot 
kinematics, the weights are previously known, and such a system needs not be 
flexible to adjust weights. In such a case, the synapse circuit diagram shown in 
Figure 4 can be significantly simplified. Instead of using the circuit in Figure 4 for a 
programmable weight, the circuit shown in Figure 5 with a fixed weight by adjusting 
a proper W/L ratio of the transistors, Mref and Ml, can be used. If a known weight 
is positive, then the /7-channel current mirror (M2 and M3) is necessary; otherwise 
only the n-channel current mirror (Mref and Ml) will suffice. 

SIMULATION AND LAYOUT DESIGN 

As a case study of this neural chip, a simple 2-link robot forward kinematics has been 
tested, and the functionality of the circuit has been verified with the SPICE circuit 
simulator. The layout design is based on a standard 2-uvn CMOS technology. 
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Figure 4: 8-bit digital programmable synapse circuit with a sign bit. Transistors M1-M8 are 
sized as 2/2, 4/2, 8/2, 16/2, 32/2, 64/2, 128/2, 256/2. Ail other transistors are of minimum 
size. B1-B8 and the sign bit can be programmed and controlled by an SRAM cell array. 
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Figure 5: Fixed-weight synapse cell. 
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Figure 6 shows the 2-link manipulator used in this design. Here a point mass is 
assumed for simplicity. Craig (1989) gives an excellent explanation of detail robot 
kinematics. For this particular 2-input/2-output forward kinematics design, there are 
three hidden neurons and two output neurons, and thus the number of synapse 
circuits required is seventeen (nine for the hidden layer and eight for the output layer) 
as shown in Figure 7. Some of the experimental results are listed in Table 1. As can 
be seen from this table, the results using our proposed design are promising 
compared to the desired outputs. The error is caused mainly due to the training error 
which is 0.0787 in this example. 

Since the synaptic weights can be programmed via an SRAM cell array, some 
other applications that have the same network architecture as this example can be 
implemented as well. The entire area required for the layout design is approximately 

3200nvnx3200|-im. 




Figure 6: 2-lfnk planar robot manipulator. Point mass is assumed for simplicity. 




Figure 7: Two-layer neural network structure for the 2-link robot forward kinematics example. 
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Table 1. Circuit simulation results of some experimental forward kinematics computation. 
(Average sum- of -square error of the network training in this experiment Is 0.0787.) 
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CONCLUSIONS 

VLSI architecture for a subthreshold operation of a programmable, general-purpose 
neural network has been discussed in this paper. Virtually, only nine MOS transistors 
are necessary to produce a positive sigmoid function in this proposed design. As 
described in the previous section, the experimental results of the forward kinematics 
example using the proposed circuit structure are promising, and such a practical 
system can be implemented in real-time. 

The approach that has been described in this paper is not the sole solution for 
realizing a fairly large, practically useful artificially neural network in the near future. 
It may be that the total number of neurons and synapses in the network should not be 
limited to certain values. In case of large network systems, a wafer-scale integration, 
or other digital approach, such as using DSP boards, may be powerful and useful. 
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